Last week I'm having dinner at this restaurant and
I'm looking around. The place is packed, everyone's getting their
orders on time and I couldn't help but think about the logistics that go
into a restaurant, turning raw ingredients
into these delicious meals. Let's think about
this for a minute. In a commercial kitchen, we have raw ingredients
being delivered by trucks to our loading
dock on large palettes. Truck comes in to
the loading dock, they drop off the palette
and the truck is back out on the road to deliver more ingredients to
other restaurants. That's the easy part. Now, we actually have to unwrap this palette
and process it. We have to sort
everything on it. We have to label all
of our ingredients. Then we also have
to make sure that each item is routed to
the correct storage area. These things could be going
into a pantry for dry goods, or it could also be
going into large walk-in fridges and freezers for things like fresh vegetables and meats, and we also have to organize
those storage areas. We've got to make sure
that ingredients that are expiring first
are used first. We've got to make sure
certain ingredients are separated from one another
for contamination reasons. We also have to make sure that certain ingredients hit a
very certain temperature, also for food safety. By the way, we need
to do all of this as quickly as possible. To minimize things
like food waste, to minimize spoilage that we could see from the ingredients just sitting
on the truck or on a pallet. Without this process,
the cooks in the kitchen can't really do their job as
effectively or safely. They'd be spending a
lot of their time just looking for ingredients
and less time actually cooking and serving out meals to their customers. What does this have
to do with data? Well, if we think about it, this very same process
also exists within data architectures
of organizations. You've got all sorts of
different data coming into your organization
from different sources, such as in different
Cloud environments, different operational
applications. Now we even have
social media data. All this is coming into
our organization just like a kitchen has ingredients coming from different suppliers. Constantly have data coming in, we need a quick place to
dump all different types of data in different
formats for later use. We have data lakes. Now these lakes allow us to cheaply and quickly capture raw, structured and unstructured
and even semi-structured data. Now just like in the kitchen, we're not really cooking
on the loading dock. Now maybe I can put a tiny grill there if I really wanted to, but we have to organize and
transform this data from its raw state into
something that's usable for the insights and analytics that are business
wants to generate. We have enterprise data
warehouses or EDW's, where data is loaded in, sometimes from a data lake, but sometimes from
other sources like operational
applications and it's optimized and organized to run very specific analytical tasks. Now this could be powering different business
intelligence or BI workloads such as building
dashboards and reports. Or it could be feeding into
other analytical tools. Just like our pantries
and freezers, data in the warehouse
is cleaned, organized, governed, and should
be trusted for integrity. What are some of the challenges that we see in this approach? Well, as we said, data lakes are really
awesome to capture tons of data in a
cost-effective way, but we run into challenges with data governance
and data quality. A lot of times these data
lakes can become data swamps. This happens when there's a
lot of duplicate, inaccurate, or incomplete data, making it difficult to track
and manage assets. If you think about it, what happens when that
data becomes stale? Well, it loses its value
and creating insights. The same way that
ingredients go bad over time in our restaurant
if we don't use them. Data lakes also have challenges with query performance
since they're not built and optimized to handle the complex
analytical queries. It can sometimes be tough to get insights out of lakes directly. Let's take a look at
the data warehouse now. Now these are really great
at query performance. They're exceptional, but they
can come at a high cost. Just like those big freezers
can be very costly to run, we can't put everything
into a data warehouse. Now they can be
better optimized to maintain data
governance and quality, but they have
limited support for semi-structured and
unstructured data sources. By the way, the ones that
are growing the most, that are coming into
our organization. They can also sometimes be too slow for certain
types of applications that require the freshest data because it takes time to sort, clean and load data
into the warehouse. What do we do here?
Well, developers took a step back and said, "Hey, let's take the best
of both data lakes and data warehouses
and combine them into a new technology called
the data lake house." We get the flexibility and we get the cost-effectiveness
of a data lake and we get the performance and structure of
a data warehouse. We'll talk more
specifically about the architecture of a data
lake house in a future video, but from a value point of view, the lake house lets
us store data from the exploding number
of new sources in a low-cost way and
then leverages built-in data management
and governance leaders to allow us to power both business intelligence and high performance machine
learning workloads quickly. There are plenty of ways that we can start
using a lake house. We can modernize our
existing data lakes. We can compliment our data
warehouses to support some of these new types of AI and machine learning
driven workloads. But we'll also talk about
that in the future video. The next time you're
at a restaurant, I hope you think about how the meal on your
plate got there and the steps the ingredients
took to go from the kitchen to the
meal on your plate. Thank you. If you liked this video and want
to see more like it, please like and subscribe. If you have questions, please drop them in
the comments below.